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The Albuquerque (New Mexico) Public Schools conducted 
a three-year study of integrated computer-based learning systems, 
including WICAT, Dolphin, PLATO, CCC, and DEGQf. Arougfa cooperation 
with the Education Consolidation Improvement Act Chapter 1 program, 
four large integrated learning systems (ILS) were purchased and 
studied. They were installed and iiQ>lemented in 1984. Due to budget 
constraints, the PLATO program was dropped trom the study. The DEGEM 
system will be evaluated during the second year of the project and is 
not included here. The primary application of the ILS was to provide 
Chapter 1 students with supplemental instruction in reading, language 
arts, and mathematics through coo^uter assisted instruction (CAI)a 
The evaluation compared the four systems* effectiveness. Aspects of 
the evaluation included the match between ILS, curriculum content, 
and standardized achievement test content; instructional and 
technical characteristics; documentation; management systems; 
diagnostic placement; programming; staffing and training; cost 
effectiveness; and student and teacher perceptions of value. Results 
indicated the CAI programs were under-utilized during their first 
year, largely due to staffing patterns. Mathematics inqpact was 
generally greater than reading. The importance of a full-time 
laboratory operator was repeatedly noted. (GDC) 



* Reproductions supplied by EDRS are the best that can be made * 

* from the original docuKnt. * 
*********** *******************^^*********<i ****************************** 



€A/: A MODEL FOR THE COHPARISOM 
AMD SELECTION OF IMTEORATED LEARNINO 
SrSTEHSIHLAROE SCHOOL DISTRICTS 



U.S. DEPARTMENT OF EOilCAT^N 
Office of Educational Research and improvement 

EDUCATIONAL RESOURCES (NFORMATJON 
CENTER (ERIC) 



Paul E. Resta. Director 

Center for Technolog/ & Education 

University of New Mexico; 

Special Assistant for Educational Computing. 

Albuquerque Public Sc^ls 



This document has been reproduced as 
received from the person Or organization 
Originatino it 



□ Minor Changes have been made to improve 
reproduction quality 



• Points ot view or opinions stated in this dOCu 
ment do not necessarily represent official 
OERt position or pol«cy 



Paul Rost, Coordinator 
Chapter 1 Computer System 
Albuquerque Public Schools 



"PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



INTRODUCTIOH 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC)." 



School districts are faced with Increasingly difficult decisions In the selection of large 
Integrated, computer-based Instructional systems based on their curriculum relevance, cost 
and educational effectiveness. Such decisions are often made based solely on the information 
provided by venders rather than a careful assessment of the differential capabilities, 
performance characteristics and educational outcomes of the systems. 

This paper describes the approach used by the Albuquerque Public Schools in conducting a three 
year study to compare, evaluate and select among integrated computer-based learning systems 
(including WIGAT, Dolphin, PLATO, CCCand DE6EM. 



In 1983, the City of Albuquerque passed a $5,000,000 bond issue designed to provide Kcess to 
computer education for all children In the Albuquerque Public Schools. The program called for a 
three-year implementation plan culminating with the addition of at least one 
1 5-m1crocomputer lab to each of the district's 1 20 schools. 



PROGRAM OVERVIEW 
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At present, all students, K- 12, are provided with a l<nowledge of computer applications, an 
awareness of the impact of computers on society and basic sicills in using the computer as a tool. 
Additional microcomputers have also been placed In many schools to address special educational 
needs. The district currently has approximately 3,000 microcomputers in place and in 
operation for instructional use. 

While computer assisted Instruction (CAI) Is clearly one cf the applications addressed by the 
district's microcomputer program , a more extensive Initiative was taken in this area during the 
1 983-84 school year. Through cooperation with the Chapter 1 program , a comprehensive plan 
was undertalcen to purchase end study four large integrated learning systems (ILS), including 
PI^TO, WICAT, CCC and Dolphin. These systems were selected because they seemed to be the 
major integrated learning systems on the market at the time. The Chapter 1 program had been 
using an older version of the CCC program since 1 976, so that system was upgraded at the time 
the other systems were purchased 

The programs were installed in the spring of 1 984 and implemented in the fall of 1 984. Since 
Chapter 1 funds were used to purchase the systems, they were placed in Chapter 1 schools with 
the greatest academic needs, a procedure followed for the allocation of Chapter 1 resources in the 
district Due to budget constraints anticipated for 1 985-86, it was necessary to discontinue one 
of the programs after the first year. Because the PLATO program was already limited in size and 
scope relative to the other three systems, it was decided to drop PLATO from the evaluation 
stud/. 

Althouph not assigned to Chapter 1 use, a DEGEM integrated learning system was placed in one 
school during the spring of 1 985. Data for the DEGEM system is not available for this initial 



phose of the evoluotion stucJ/, but will be included in the second year of the project. Table 1 
provides a summary of the number of sites and type of system configuration used during the 
first year of the evaluation study. 

The primary intent of the application of ILS to the participating schools is to provide Chapter 1 
students with supplemental instruction in reading, language arts and math through computer 
assisted instruction. Secondarily, the comparative effectiveness of the four systems is being 
studied in order that the district might be better prepared to make decisions regarding expansion 
of CAI through integrated learning systems in the future. 

ILS EVALUATION STUDY COMPQMFNTS 

The three-year stud/ has been designed to focus on the important instructional, technical and 
performance characteristics of the ILS packages, the conditions for their effective use and the 
correlation of system educational content with district curriculum. The following Is a summary 
of the major aspects of integrated learning systems that are incorporated into the three--year 
stud/: 

]. Educational Content 

a. Match between ILS and district curriculum structure and content; 

b. Match between ILS and standardized test structure and content. 



TABLE 1 

1 984-85 CHAPTER 1 CAI CONFieUPATION 



PROORAfI 



SCHOOL 



EOUIPMENT 



SETTIN9 



ccc 



Dolphin 



WICAT 



PLATO 



DEGEM 



B 



0 
E 



H 



K 
L 
M 



16 Terminals 
16 Headsets 

1 Mincomputer 

1 Printer 

1 Digital Speech 
System 

16 Terminals 
1 Printer 

12 Terminals 
1 Printer 

12 Terminals 
1 Printer 

8 Terminals 
1 Printer 

12 Terminals 
1 Minicomputer 
1 Printer 

12 Terminals 
1 Minicomputer 
1 Printer 

1 2 Terminals 
1 Minicomputer 
1 Printer 

12 Terminals 
1 Minicomputer 
1 Printer 

30 Terminals 
1 Minicomputer 
1 80 megabyte 
expansion unit 
1 Master Terminal 
1 Printer 

7 Microcomputers 

7 Microcomputers 

32 Terminals 



Lab 



Lab 



Lab 



Lab 



Lab 



Lab 



Lab 



Lab 



Lab 



Lab 



Classroom 
Classroom 
Lab 



I LS Instructional Characteristics 
0. Inslructionol range 

1) grade level 

2) ability level 

b. Program Uses 

1) drill or practice 

2) tutorial 

3) simulation 

4) Instructional gaming 
5; problem solving 

6) Informational 

7) tool use (word processing, spreadsheet , etc.) 

8) teacher utilities 

c. Lesson length (average time) 

Documentation/Resource Materials 

Mgnagement. Record Keeping and Reporting Systems 

Diaanostic Placfiment/Evaluation 

1 ) diagnostic placement in appropriate program module or lesson 

2) pre and post testing 



Programmino Capability 



7, Authoring System 



8. Micro Capability 

9. Technical Characteristics 

a. use of graphics 

b. use of color 
& use of audio 

d use of animation 

e. student interface 

f. variable tlmlng/rale of presentation 

g. effective personalization 

h. random generation 

i. user control 
]. early exit 

k. add or change content 
1. feedback 

10. Staffing Requirements 

11. Training 

a. operator /teacher training requirements 

b. training assistance provided by vendor 




12. Hordware/Softwore MointenQnce ond Service 



13. Contract and W arranty Terms and Conditions 

H. Cost Considerations 

a. lease/purchase 

b. operational expenses (operator, supplies, telephone, etc.) 

15. Educational Value ond Imooct 

a pre-test/post-test comparisons 

b. standardized test measures 

c. gains per hour of use 

16. Stu dent Perceptions of Value of Program 

17. Te acher Perceptions of Value of Program 

6ENERAL CHARACTERISTICS OF INTE6RATEP LEARNIN6 SYSTEMS 

An Integrated learning system is typically defined as a completely computerized education 
package in which thje hardware and courseware components are intended for exclusive use with 
one another (Dudley, 1983). All of the systems Included in the APS evaluation study were 
originally based on powerful minicomputers or mainframes. Although, almost all of the 
oompanies now also provide versions of their courseware for use with microcomputers, they 
indicate continued growth in the use of their ILS packages by the schools. 
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All of the firms offering ILS packages have supported an expensive and long-term R&D effort in 
producsng the systems. The following tables preside a summary of the major technical features 
(Taf 5 2) and the instructional aspects (Table 3) of the five systems that were or are presently 
included in the APS evaluation study. 

EYALtfA-ION t)E3l6N 

The first year eva'suation plan had four components: 

1 ) analyze pre- post achievement test data In relation to amount of time on CAI ; 

2) survey cla^oom teachers pertaining to the effectiveness of CAI as a tool to 
support regular classroom instruction; 

3) determine the correlation (not statistical) between the district's reading and math 
curricula and the instructional content of teach of the integrated learning systems and 

4) review, compare and contrast the management systems of the WI(iAT. OCC, and Dolphin 
packages. 

IMPACT DATA 

With only a few exceptions, the CAI programs were under-utilized in 1984-85, largely due to 
Inconsistent staffing patterns across the three programs. The original implementation plan 
called for Chapter 1 to provide the hardware and courseware for each lab and for the district to 
provide lab supervision, mostly in the form of the regular classroom teacher accompanying 
students to the lab. Each school was responsible for arranging for lab supervision. While there 
was a plan for supervision and a student schedule for each lab, the press of other matters often 
precluded Id) attendance. 
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In addition to attendance problems, a review of the time-on-task information collected indicated 
that for merry students the systems were inappropriqotely used, the result ijeing that Ihe 
students failed to experience the curriculum as It was designed. System reports indicated that 

9 

Students would exit one lesson prior to completing U. begin a second lesson and do the same thing 
again , all within the 55-20 minute session. This, of course, interrupted any continuity and 
reinforcing aspects of the curriculum. Consequently, many students were enrolled in the CAI 
program anywhere from 25-30 weeks, but accumulated relatively little constructive 
time-on-task. Both the attendance difficulty and the inappropriate use of the systems clearly 
stem from the structure of the lab supervision. Where there was stable supervision, problems 
were less evident. Table 4 reflects average hours of CAt time-on-task in supervised ( 1 FTE or 
.5 FTE lab operator) and unsupervised labs In both reading and math. Means were calculated on 
students with 5 or more hours of CAI service. 

At best, implementation and operation of the CAI programs were inconsistent in the case of some 
of theCCC and Dolphin labs. This fact makes It difficult to Interpret the available achievement 
Impact data Since the service problems w^e not related to ttie CAI systems, no comparisions of 
program effectiveness can legitimately be made. Several overview statements can be made in the 
cases where sufficient service was provided, however: 

1 ) math Impact was consistently greater than reading 

2) considerably more time-on-task was provided where there was a full-time lab operator 

3) considerably more students were served where there was a full-time lab operator 

cAi/pgRBicuiun match 

A primary concern in the seledion of dn integrated learning system is the extent to which a 
district's curriculum matches the CW curriojlum in itte areas to be covered While this stud/ 
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MUMBER OF STUDENTS SERVED AND AVERAGE HOURS OF CAI TIME-ON- TASK 



SUPFRVISED LABS 




PrfYiram 


Number 

1 Y W III Wl 

Served 
Math 


Averaoe 
Hours 
Math 


Number 

Served 

Readina 


Average 
Hours 
Readina 


F 


Dolphin 


95 


32 


70 


36 


G 


Dolphin 


160 


31 


104 


12 


H 


Dolphin 


1 


29 


64 


15 


1 


Dolphin 


0 


0 


90 


NA« 


J 


WICAT 




24 


84 

V I 


21 


E 


CCC(.5FTE) 




17 


31 


9 
✓ 


UNSUPERVISED LABS 










C 


CCC 


9 


6 


0 


0 


6 


CCC 


38 


10 


26 


11 


A 


CCC 


48 


8 


16 


9 


D 


CCC 


34 


10 


2 


5 



* NO HOURLY DATA COLLECTED AT -SCHOOL "1 " DUE TO LATE START. 
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was not done prior to implementation, the results are still proving useful Three classroom 
teachers and three Chapter 1 reading teachers were asked to compare the district's reading and 
math Instructional objectives with the three CAI systems to be continued Into the 1985-86 
school year - Dolphin. WICAT and CCC. Each reviewer had considerable experience with the 
particular CAI systems and the subject area they were stud/ing. 

Only the elementary portions ( K-5) of the CAI curricula were studied. Comparisons were made 
in the areas of math and math problem solving, reading and reading comprehension. 

As shown on Tables 5 and 6 . each program had its own strengths in the comparisons: 

1 ) WICAT compared most favorably to the dIstrtcVs curriculum In math, covering about 
of the instructional objectives across grades 1-5. 

2) Dolphin compared most favorably in reading, covering almost 80SB of thQ^d[strict's 
reading objectives at grades 3-5 (both CCC and WICAT offer reading curriculum at grades 
K-2. but that comparison has not been completed). 

3) WICAT compared most favorably to the district's reading comprehension program across 
grades 3-5 by covering 90^ of the instructional objectives (CCC and Dolphin also 
compared favorably, covering 82^ and 77^ of the objectives, respectively). 

4) CCC provided a more consistent match In each area by grade level, although wide 
variations in the match from one grade to another were noted in all systems. 

However, the results across all three systems and subject areas (reading, math and reading 
comprehension) were very similar across grade levels: Dolphin. 67^ match; WICAT. 63^; 
and CCC. 6Z% match. It is not clear that these comparisons are either good or bad. Additional 
impact data will be necessary in order to make that judgment. 

ERIC 



TABLE 5 

lis CURRICULUM AHD APS CURRICULUM MATCH BY LEVEL 

MATH 

DOLPHIN WICAT CCC 



LEVEL 1 27% 47X 332 

2 25Z 38S 382 

3 502 502 392 

4 632 632 502 

5 542 712 572 

ALL 442 542 432 
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TABLE 6 



lis CURRICULUM AND APS CURRICUl UM MATCH BY LEVEL 
COMPREHENSION 
DOLPHIN WICAT 000 



LEVEL K 25% 

1 100Z 

2 BOS 50% 

3 672 922 752 

4 852 922 922 

5 802 872 812 

ALL 772 792 752 
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CAI/ACHIEVEMEMT fEST MATCH 



A similar analysis Is being performed between the content of the ILS and the major evaluation 
instrument used by the district. Tables 7 and 8 are two examples of the analysis tables 
comparing content of the CTBS with the ILS curricul;jm. At presant, the analysis has only been 
completed for two grade levels. 

MANA6FMFNT COMPOMENTS 

Another Issue of major significance in the selection of an integrated learning system is the type 
of management system available with the CAI program. To a large extent, the manner In which a 
school or district elects to staff its CAI program will determine whether the management system 
will be effective. 

The CCC management system appears to be the only one that permits a CAI lab to operate without 
a full-time lab operator. Even with this capability, the CCC program functions much more 
effectively with a lab operator. WICAT and Dolphin definitely need to have lab operators to run 
their programs. 

All three programs have initial placement procedures that facilitate implementation for 
students. CCC and WICAT placement procedures are more automated than Dolphin's. In addition, 
progress through the curriculum is largely automatic (machine driven) with WICAT and CCC, 
while Dolphin requires ongoing manual input to keep a student moving through a curriculum 
area. 
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■■S CURRICUHIM & CreS MATH COMPUTATIOII COIITEIIT HATCH - GRADE 4 



.sm POUHW ^ Vff^T 



ADD 

WHOLE NUMBERS 



ADD 

DECIMALS OR FRACTIONS 4 



SUBTRACT 
WHOLE NUMBERS 



SUBTRACT 
DECIMALS OR FRACTIONS 4 



MULTIPLY 
WHOLE NUMBERS 



8 



DIVIDE 
WHOLE NUMBERS 



MATH CONCEPTS 



NUMERATION 
NUMBER SENTENCE 
NUMBER THEORY 
PROBLEM SOLVING 
MEASUREMENT 
6E0METRY 



10 
7 
6 
9 
9 
4 



X 
X 
X 



X 
X 
X 
X 
X 



X 
X 



TAP L E 9 

ILS CURRICULUh & CTBS I ^^ BUAGE ARTS COWTENT MATCH - GRADE 4 



CTBS 



CAP OF PRONOUN I 

NOUNS & ADJECTIVES 6 

CAP OF BEOlNNtKB 

WORDS & TITLES 4 

USE OF PUNCTUATION 

MARKS 8 

PUNCTUATION & 

CAPITALIZATION/EDIT 6 

NOUN USA6E 4 

PRONOUN USA6E 4 

VERB USAGE 5 

ADJECTIVE/ADVERB USAGE 4 

SENTENCE PATTERNS 

SENTENCE FORMATION 4 

SENTENCE RECOGNITION 4 

SENTENCE COMBINING 10 

TOPIC SENTENCE 5 

SEQUENCE 5 

USE OF QUOTATION MARKS 4 



DOLPHIN CCC WICAT 



XXX 

XXX 

XXX 

X X 
X X 
XXX 
XXX 
X X 



X 

X X 

X 
X 

X X 
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However, with a full-time lab operator, placement and progress issues do not have to be 
stumbling blocks with any of the programs. 

There are more similarities than differences among the three programs in the areas of setting 
up classes, enrolling students and changing or adding courses or assignments. Again, a full-time 
lab operator ameliorates problems associated with these activities under a structure of less 
supervision. 

Student progress reports are the link between CAI and the classroom instruction it supports. 
While all three CAI systems have extensive reporting procedures, the reports may not be equal 
In their useablllty and readability. Reports were not used consistently with any of the programs 
in 1984-85. so data regarding their appropriateness are not available. The overall CAI 
configuration of 1 985-86 will allow for extensive use of the student and class reports and their 
appropriateness will be carefully studied. 

The CAI report features considered important at this point are: 

1 ) indications of student growth or progress in a form or metric familiar to teachers; 

2) relative relative strengths and weaknesses in the subject areas in which the students are 
enrol Ujd, and 

3) progress in terms of skills mastered. 

With these data, classroom teachers can optimize the impact of CAI on participating students. 




19 



11 



CLASSROOM TEACHER ASSESSMENT 



In addition to traditional impact data such as achievement test scores, a large numfcer of teachers 
(75) in the CAI schools were asked to assess the effects of their respective CAI systems on their 
students. In general, teechers were very positive. They observed that students made more 
progress in math than in reading. This progress was most often reported as showing up in the 
students' dally classroom work. Responding teachers did feel that CAI can serve as an Important 
supplement to instruction for Chapter 1 students. 

RECOMMENDATIONS 

After one year, a great deal has been learned about CAI. Based on the results of the first year 
evaluation stud/, the foliating points were taken into account as the CAI program continued 
during the second year: 

* To guarantee maximum usage of the systems, consideration should be given to funding an 
operator Tor each lab; 

* To facilitate meaningful communication, utilization and integration throughout the year, 
classroom teachers should continue to be Included In training In the currlcular content of 
the systems; 

* To assure appropriate expansion and Integration , the stud/ of CAI management components 
and cirrlcula content should continue, and 
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* To insure proper interpretation of 1985-86 impact data, the content match between 
the achievement test (CTBS) administered at Chapter 1 schools and 
the curricula of each of the three integrated learning systems (CCC. WICAT, Dolphin) in 
use should be completed during the se<:ond phase of the stucV. 

Based on the results of the first year of the evaluation study, it is clear that a number of 
conditions must De estaoiisned m oroer to maice a more realistic ana comprenensive assessment 
of the relative advantages of the different ILS packages for continued implementation in the 
district. During the second year of the study (currently in progress) such conditions have been 
established and usta will be obtained for the data categories identified in the model. 
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